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1. UJTRODUCTION 



This final report, sumnarize 



/. 

' '1' ' ' 
;es-^ar 11 qir 



quality c;}ieck activities for the 



NatiOMl AsSessMttt of Educational Progress (NAEP)i A probability sample 
of 40 schools was selected for quality check purposes from all three age 
classes. *Oiie regular school wa« selected for each District Supervisor at 
each Age Class. Quality check acta^vities were conducted in these schools 
during the perigd of October 1979 through May 1980 by staff from the Re- 
search Triangle Institute (RTI). All checks were conducted with, the con- 

■ 

sent and cooperation of the schools involved. 

' ' ' : 

The sample design and other sampling activities associated with the 
Year 11 quality check are documented in section 2 of this report. The 
procedures used to implement the quality check are discussed in sectioa 3, 
^ The resullis of the quality check are reported in section 4. Conclusions 
'and recommendations for future quality ct\ecks are presented in section 5* 
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2. SAMPLE SEiECTION 



p « 



2,1 School Saaple Selection - • ^ 

The Year 11 quality check school saiuple was designed to meet the 
following specifications: 

(A) At least one regular (nonstandby) school was selected per age 
class for each Bistrict Supervisor; * ^ 

(B) A ratio estiiuate of the conxpleteness of the student sampling 
frame could be obtained across all age classes; ^ 

(C) An estimate of the variance of the "ratio estimate in item B above 
cduld .be obtained. ' ' 

Table 2-1 summarizes thH'^thod of sample sele'ction* The frame of 

regulaj; schools was stratified by District Supervisor assignment, yielding 

13 strata* Within eaclj District Supervisor stratum, schools we^ further 

stratified by age class* Withi-n each District Supervisor by age cl^^s 

stratuiii""^^e regular (nonstandby) schools comprised the sampling units, 

with two exceptions. First, each standby school located within. 20 mil^s of 

a regular school>vWa5 grouped with the regular school to comprise one sabpl- 

4 

* ing unit* Second/ split and modular sessicia schpols we^e grouped witi the 
♦ 

originally selected school as one sampling unit* The number of Sampling 

* » 

units in each stratum is specified in table 2-1* One siampling unit was 
selected per age class for each District f Supervisor stratum* The units 
were chosen with, equal proU^KLlities within strata. By weighting the data 

for Mch sample unit by the inverse of its selection probability, the 

* * 

completeness of the student frame may be estimated across all age classes. ^ 
^ estimate of the variance <yf this ratio estimate may be obtained by 
taking s*iccessii/e differences between sampling units within District Super- 
visor strata.; _ 



Uhlc 2-1. QuftUty ckeck lakpU leUc^ioo^ lu^^ry 
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395 
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Co*|>ri«cd ^1 one rctuUr «ctiool «ad oa« standby «cbool. 
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By these procedures, a probability sample of ' 40 schools *C39 regular ^ 
schools and one standby school) was selected. A list of .these schools is 

• • 5 ^ V 

Included as Appendix A" Table 2-2 below summarizes the sample by 'region 
' and age class*^ ^ • * • 

2.2 Student Sample Selection , ' ^ 

A random Sample of packages in each quality check school fas selected. 
If five or fewer administrations were assigned, all packages were. i^cludfed. 

If the number of assigned adminisljuations was greater than' five, these ad- 

♦ 

ministrations were subsampled to five using simple random sampling^ Three 
respondents from each of^the packages in the sample were selected using 
simple random sampling tecimiaues ■ The students were interviewed insure 
that they had taken packages. , ^ 

A sequential saaipling procedure was used to verify that the student 
sample in each school had been property selected A A table* like the <3ne 
^shown in figure 2-1 was provided for each school. The sequential sampling 
pfoceis was as foliowj,; ,^ ^ * 



(A) The first five in^range random numbers w^re examined.. 



Table^2-2; Number of quality check sxrhools by region and age class 



Region 


9.-y ear-olds 


13-year-olds 


17 -year-olds 


1 

Total 


Northeast 


3 


S 


3 


-11 


Southeast 
* 


3 


* 

2 




\' 


Central , < 


4 


'** 4* 


3 




West 


_3 


■' J, ' 


_3 


_9 


Total' 


. 13 ■ 




13 


*A0* 



IncludesJon'e standby school. 
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SAMPLE ^ELECTION dkECKING PROCEDURES 



PSU Nufflber , > • 

» « 

School Number 



? ' -".^ » 



Dlrectionsi Examine siat>le 'selection, cackle assisramenc. and tratiscribln« 
orocedurea for first 5 selected resoondents. Document each tvne of error 
in snaca provldfed below. Record mdsber of, errors on' first line of "Cumulated 
Errors column. If the number of cumulated errors is less than or equal to * 
corresponding accept^ce number, then stop. Otherwise, examine the method 
used to select the next 5 individuals, pocument the types of errors. Record 
number of cumulated jsrr<vrs . If the number of cumulated en^ors is less th^n 
or equal to. the corresponding acceptance n<nnb6r,' then scop. Otherwise, 
examine the method-used to.seleC^ the "ne^t 5 individuals, etc. Stop, after . 
nechod usitd to tf^lect at most 20 respondents: has been examined. 



Cumftlated ' Cumulated. Acceptance 

sample size ^ • errors number 



. 5 
10 
15 



-r 

V 

0 

2 * 

4 



20 ' ' fit 



Types of Errors 
V 



-J 



4 



/ 



^Figure 2-l» QuaUcy check, form td assist iii 

sequent iiL sampling process v 



(BJ The, number o£ incorrectly used random numbers out of the five was 
recorded in the cumulated errors c'oluma (column 2)^ 

(C) ^ If this number was zero, .it was not necessary* to checjc any more 

random numbers j[i.e,, the sampling pcocess was assumed to "be 

cprrect) . If the number was one or greater, the types of errprs 

^ were documented and the next five random numbers were examined. 
• • • 

' (D) The cumulative number of errors in the first 10 random numbers 
wlks recorded in ,co],upa 2. As before, if the total number of* 
errors was less than or equal to the number upecjLfied in column 3 
^Acceptance numb^er^* the samplp was assumed to be correct, and no 
" further checks were made^ if th6 total number of etiors was 
greater than' the number specified in coXuma 3, errors were docu- 
mented and the next five random numbers were examined. 

. (E) This process (^was continued until a decision was made as to 
whether tfhe sample was acceptable or unacceptable. At most, 20 
random numbets were examined. If six or fewer incorrectly used 
randoili numbers were identified among|i,the 20, the sample was 
acceptable. If more than six incarri^t^y used i^ndom numbers 
were identified, the sample was unacceptaole; in either cas^, all 
errors were documented. ^ 

Table 2-3 specifies the probabilities of accepting or rejecting the 

sample, after examining each s^t of five random numbers, assuming that the 

sample has been correctly selected 50 and 90 percent of the 

* * • ... I 




r 



Table 2^3; Probabilities of accepting and rejecting sample after 
examining each set of -five/random numbers 



Cumul; 
sampl^ 


* 

[ted 
^size 


Accept*ance 
nlimber' ^ 


Probability of 
acceptiMs sample 


^ Probability of 
reiectin^ sample 


.P = .50 


p'= .90^. 


..p = .50 


p = .90 


. , ..5 




' 0 


0.03U.* 


0^5905 


o.ooob 


o.qooo 


10 




2 ' ■ 


• 0.0547 


0.9298 


" 0.0000 . 


0 .0000 


' .45 




4 


0.0593 


0,.9873 ^ 


0.0000 


0.0000 


20 


* 


. 6 


, 0.0577 


0.9976 

• 

CjU 


0.9423 


0.0024 



,2 , 3 Verifying a Subsample of thie Student 

■ . - " ~ ? >^ » ' . 

In quality check schooli| *the stndent list was examined for complete- 
ness. pii^' examination was perioroed by pre^ari^g^a complete student li^t 
from the school records and ^comparing the list with the^ one prepared by Xl^e 
school. In. some cases the. most complete stuc^^nt list 'was the'*one prepared 
by the schoojl^- Jn other cases a better Jt^t was prepared during the qua- 
■ lity check. , ^ - • ^ /" , , ^ - ^ 

/ Xf the' school, contained m^ny eligibles,^ a check f or ^completeness was 
txmt consuming. Whe;^ tijere wete more th?ti 200 ,eligibles ,in a school, the 
optaon of selecting a sub^mple .oi 200 was permitted. ' An alphabet^^ 
ment was de^ermike^d ^and recorded (see i^pendix Q fpr qual^^l^ check forms). 
The check was pe;f&rmed only for students in the alphabetic sector. The 

aljiWbetic segments were determined using a proceSure" developed l>y Piper 

\ ^ : ■ . ' - ^ - 

and Chromy (iL Table 2-4 lists the alphabetic segments which were used to 



achieve various sampling ratfss. 



t 



J 



m • ' 



-3- 



. i Table 2-4. Appr&klmaCQly equ^*-size'd alphabetic segments 



1 



Ali>habetle 
sement 
•ifeer 



' Approidinate proportion ofr names eootalned In each secaent 
1736 I 1/18 I- 1/12 LMy9 i 1/6* .1 V3 I 1/4 




1 . 
2 
3 
4 

. 5 • 
6 

7 . • 
8 

10 ' 
- 11 
12 • 
" 13 • 
* ' i4 

15 - 
16 
17 > 

'19 

' ; 20 

' "^'21 
' ' 22 

24 
< 25 . 
26 

28 ; 

29- 
30 
31 

'33 . 
34, 
35 • 
36 



AAA-ASH 

BAS-BLZ 
BHA-BRO 
BRP-CAQ 

COA-CRD' 
C^-DED 
DEE-DZZ 

EAA-PEZ/^fSte^FEZ 
FFA-GEN 
GEO-GZZ 
HAA-HAZ. 
HATf-HOK 
HOL-HZZ 
lAA-JOH 
JOI-KEK 

LAA-LIS 
LIT-HAR 
liAS^KDZ 
MEA-MON 
MO%-NAX 
HAJ-OZZ 
PAA-PIN 
PI0-RA2- 
R^-RZZ 
SAA-iEA 

'SIR-SNZ 
•SOArStQ 
STR-THN 
TH0-U2Z 
VAArWER 
WES-47IL 
OlQ^Z^Z 
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' - 3. QUALITY CHECK PROCEDURES 

A quality ' check was ^ondiTcted during the period of October 1979 
through May 1980. The check was designed to determine the quality of the 
Year 11 assessment data collected by th) National Assessment in*school 
field spaff . -The quality check tas?ks conducted were la^jgrfy a review of 
the mechanical and clerical perfo^nances of school personnel, Distifict 
Supervisors y and Exercise Administrators in^lve4 with assessment. 

^ The design of the Year 11 quality check was derived from discussions 
between members of the National Assessment of Educational Pirogress (NAEP) 

k 

Staffs, RTI's National Assessment Adminirstration Center (NAAC) , and RTI's 
Sampling Research and Design Center (SRDC) . The check was Jjuplemented by 
the National Field Director, Regional Supervisors, and Administrative 
Coordinator from RTI*s National Assess^nt Administration Center. All 
checks were conducted with the coftnnt and cooperation of the schools 

V 

involved. % V, 



3» I Sample ^ ' ' ^ ^ 

A1l{)robability sample of AO schools was seledted by RTI's Sampling 

^ Research and Design Center. The sample included schools from each region 

of the country and one. school per a^e class from each District Supervisor'^ 

t • 
Xerritpr^i. The sample design is documented in section 2.1. 

The sample design for the .quality check allows inferences to be made 

concerning the nationwide quality of work done in Year 11 assessments The 

sample'^w^ not designed tro yield information^ on the performances of indi- 

vldual District SuperV|isors nor was it. designed to yield this informatiop 

by region of. the country,^ , ^ . 



"3.2 Quality Check Tasks ' ' • , ' 

Several tasks ifere perfomed in each of the, saoiple schools. Each 
school was contacted y aM the tasks were described to the principal and/or 
assessment coordinator* Permissiotv was given fox; the qua l>fey^ check to be 
perfonn^ in all 'schools. The attitude demonstrated by the majority of 
school yofficials contacted was cooperative. ^ 

jitter arriving at the ^fiooX, the school principal or coordin^ttor was 
ask<^ to prdvide theiStujlent^isting Form (SLF)' Storage Envelope containing 
the SLF and the Administration SchedutfeCs) left by the District Supervisor. 
The purpose of this i^^quest was to permit the sampling procedures performed 
by the District Superyi'&or to be checked. It was also explained that a 
sample of three randomly selected students who had participated in assesp- 
ment would be interviewed from at most five of the regolar group sesbions 
assigned to the school. The students were selected directly from the 

.Administration Schedules using a random number table (see Assessme^t^ Veri^ 

f ication f 3rm^ in Appendix B) . - ^ * ' 

After the ^ab<^e taslu were completed , the school official was request^ 

ed to provide a central^ record of aTl students enrolled in the school that 

contained' birthdate information. The record was us^ to determine the 

total number of elx^ble students' enrolled in the school. The^ total ^^ount 

found during the quality check wgs entered dn the Total Eligibles form (^ee 

Appendix B) • This^ number was compared with^the number ff eligibles listed 

• " • 

on ^e School Worksheet as well as the number found on the Student Listing 
Form. This task was j^esigned to check the completeness of the^report^ng, tfl' 
eligible^ students* ^ ' ' Jt 



i . • ■ •• • 

I . 4. i^miS OF SPECIFIC QUALITY CHECK TASKS 



* Specific Year 11 quality tiheck tasks included, verification of ass<^ss« 
ment respondents » determination of the completeness o^f the ,student(^ list, 
^nd verification of the student sampling procedure. t 
4.1' Assessment Verification ' 

; ' 

students * were interviewed- in sample schools where school of ficials 
granted pejnaission. The imrposes of the interview were to verify the 
'student's participation and to verify that correct procedures had been 
followed J)y the DS or The Assessment Vetification form (sde Appendix 

B) was administered to a maximum of three students pe^ group, A random 

i 

sample of tl^e regular group sessioq^ assigned to a school was preselected 
Iby RTI*s Sampling Resea/ch and Design Center- If five or fewer sessions 
Were assigned^-^^I packages were included. If the number of sessions was 
greater^,;^n five» the sessions were subsampled to five using simple random 
sampling. All nonrespondent followup sessions*^ assigned to a school* vere 
vej;ified 

' Selected student n^es bad been previously numbered on the Group 

4 

Admioistration Schedule for each session. Us^ng this schedule and a random 
number table, three student s^^f^v- package were selected to be interviewed. 
If .two or all three of jthe stud^ts selecte4 for verification of a^ session 
were unavailable because of abs^mce from school ^ withdrawal from school ^ or 
inability to leave class, replacements for those'^studetftswi^ selecte^ 
Of the 40 quality Check schools » assessment verification was C9nducted 
in 37 s^ool^. "In one S-^year-^old^ school, verification of 2nd graders was 
not conducted due to objections of the school superintendent. The prih-* 
cipal of one IS'^year^old school would not allow respondent verification. 



One school at the 17-year-old age level would not allow. respondent v^ri* 
flcation because student accessibility would be difficult due to lit being 
the last d^ ci school. In total, 386 students were interviewed in the 



V 



quaHxy check sample. This number included 115 9rytfa5-olds, 124 13-year- 
■ ■ . 

4. "V 

olds I and 147 17-year-olds. * 

• . ■ • • * / 

The results of the assessment verificatioa.ara summarized by age class 




xn table 4-1'. All st.udents Interviewed remembered participating in National 
Assessment. ^ 

One student at age class 2 said the test administrator had left th 
testing room for^ about three minutes. Another student at age class 
stated the test administrator left the testing room twice for about five 
minu^s each time. Threii 17-year-old students reported that package aminis* 
tration was begun by the school coordinator until the EA took over 10 
mlniites into the session. Another age class 3 student responded that the 
t^t administrator was not always present di^raag testing bu^/,did not give 
any explanation for the; absenc\. 

In respl)nse to question 3^ some students had observed principals^ 
teaciierSi counselors , or librarians in the room duj^ing assessment at age 
clarss 2 and 3. These individuals were monitoring or^ observing sessions and 
no irregularities vTere detected. 

6ne student at veach age level £elt the instructions were uncleat. One 
student was 10 minutes late and entered from the point at whicrh the Xape 
^was playing. Two other students repotted that the tape was hard for them 
to* hear. All other students responded that the tist instructions ^ere 
clear. 

Almost all students interviewed remembered the Background Questions 
Jbeing -administered. Foiir 13-year-olds and one 17-year-old did not 



Table 4.1 Assessment respondent verification 



Question 



Answer 



1. Do you remember participating 'in 
National Assessment? 



2. 



( 



Was a test administrator present 
throughout the test? 



Were there any teachers or a 
principal in the room during 
the testing? 



5. 



6. 



X 



Were the instructions clear so 
you knew what to 4o? 

Were you asked questions such 
as I "Does your family get a 
inewspaper regularly? 

Could you hear the^tape recorder 
frpm where you were sitting? 

.Did the person conducting the 
test help you with any ques- 
tions 'wtille the tape" was play^^g? 



^A^proximat^ly how many studeiHs 
. took Uie^ test with you? ^ 



I 
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Yes 
• Ho 
Total 

Ves 
No 

Don't Know 
Total 

Yeff 
No 

Don't Know 
Total 

Yes • . 

No, 

Total 

m 

% 

Yes 
No 

" Don't Know 
Tptal . 

Yes • . 
Ha 
- Total 



9"year-old6 


-115 


(100%) 


0 

If 




TH 


(100%) 


115 


(lOpX) 


. 0 


( m 


0 


( 0%) 


-115 


(100%) 


0 


( m 


\ 115 


(100%) 


\ 0 




115 


{looj) 



13-year-olds 



. 17-year-olds' 



lU (-99%) 

TH" {ipS^T 

115 (100%) 
0 (■• 0%) 

115 • tToo%T 



112. (.97%) 
3%) 
lOSfT- 



"3 (y 3%) 
lis (iT^ 



Yes 
Hp 

Total ' 

Acceptable 
response 

Unacceptable 
responee^c 

Don't KqoK 

Total - \ 
t 



D (• 0%) • 
,H5 (100%) 

115, TToo%y 

■ * 

. 106 ( 92%)' 

. * 2^ ( 2%) 



1^4 (100%> 
0 (. 0%) 
TlO^ 



122 \ 98%) 
2 ( 2%) 
Of 0%) 

.124 (TooiT 

• 3 ( 3%) 
121* ( 97%) 
0 ( 0%) 

12? Two%y 

123 ( 99%) 
.1 ( 1%) 

12? (Too%y 

12r '( 97%) 
•1(1%) 

124 (1 



o5%y - 



119 .( 96%) 

124 (T00%J 

( 0%) 
124 - (100%) 
124 (400%! 

122 ( 98%) 

. 0 0%) 

2 

^12? 



147 (100%) 

0 ( 0%) 

^45 •( m) . 

' 2(1%) ' 
^ ( 0%) 

. 147 ■(Too%T 

. 20 '( 14%) 
127 ( 86%) 

147 Tioo%T 

' • 

, 14"6 ( 99%) 
147 (100%). 

U6 ( 99%) 

1 •(• 1%) 
■ 0 (.0^ 

157 ^%T ' 

130 (-88%) 
Jl ( 12%) 
147 (100%> 

5 (* 3%) . 
, 142 C 97{C) 
147 (TMT 

l'4l'(96%) 
.3'r^) \ , 



U 
1 



I 



J 



remember. Most students interviewed could hear the tape- Those wl^o could 
not hear encountered audibility problems with the tape (i.e., volume too 
low or bass too loud). The tape was not used for one 17*year-*old session 
due to tape malfunctions 

The students who^responded that the pe^tson conducting"" the test helped 

them while tlib tacle wis playing noted that the help was always explanatory 

# * 
and that ^ no specific questions were answered. Seven 9-^ar^olds, two 

13-year-olds, and three 17-»year-olds did not remember how many students 

taok the test with them. Two 9-year-olds and three 17-year-olds gaVe 

unacceptable responses. All other students interviewed gave .acceptable 

responses. * # 

4.2^ Count of Eligibles 

\ 

4.2.1 Description -of the Data Collected - 

The. eligibles in schools selected for the /luality check were enumerat- 
^d again. The second count of eligible? was coinpazjed with the original 

cdunt obtained. The data from this check are enumetated m table 4*^2. The 

• \ - , > » 

PSU and school number are specified In columns 1 and 2)%respectively« The 

number of eligibles found 341 the quality cheeky is^ not^d in the third column 

(coliflnn A). The "number of eligibles recorded on the School Worksiteet^ 



noted* in Column B. The diffefl:<}nce, A-B, is reported in the following 
. . column, 'the percent dif f«^cl!**^e^ween the number of eligibles found in 
V ''the""^lity chetk and >the number recorded on the School fWo'xksheet is report- 

; , V A ^ ^ 



Tfcis percent figure is rt^ottfi^rtf£ xiolx^ 6.^ The last columr of the table 
provides a brief explanation foy large differenot^. * . 
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4*2*2 Ver^ication Results of Eliigibility Counts « 

It vas asslimedy since the age class eligibles were checked between 

four and six weeks after ^the assessment period, there w<juld be very few 

students leavi'ng the school during this period. It was also^ assumed that 

the nu4|}er entering would he approximately the same; thus, no adjustment 

•was. made for attrition* - ^ 

A Vatio estimate of the*^ proportioD^ of students reported in Year 11 

sample schools sja^ 0«984« The standao:d etror of this estimate is .0046* 

The pro,cedure by ,which this estimate was ol;(taiaed is documented in Appendix 

C* The 95 ptftcent confidence interval for the estimat^ed proportion based^ 

on 25 degre^f^f^ freedom (Student's t 2.060) gave • . . 

• , *. ' 

/.975 < R < 0.993 • 

In the preceding , inequality, the estimated proportion, ol students reporrtec 

in Year 1*1 saiftple schools was noted by R*. The cc^mpleteness of the 9-, 13*, 

and 17-year-^ld lists for three earlier years is compared in table 4-3* 

^Tajbie 4-4 was prepared by tabulating the number of large errors, it SJJ 

pi^paration by region* large errors were defined as those wher§ the absb- 

lu^i^vaiue'of the , percent difference in eligibles wa?. greater than 5 per- 

_ cent^ iy chi-Square value of 0*68 w^s calculated and compared witlf the 

Jtabuiar/ value of^ 7*81 (with 3 degrees of freedom)* It may be assumed with 

95^ P£«ent confidence that no region has more .large errors than would be 

expected bv chance* 

4*3 Verification of Student Sampling Procedure 

' ' ~ . 0 

The form 'in figure and procedure described in section 2-2 were. 

used to verify that thb Student sample was ,ct)rredtly selecK^ed ill each ^ 
^^uality check schodl* A few errors in the method of student sample selec- 
tion were dliplosed by the quality check procedires* In one, school 70 ^ 

23 ■ • • • •■ 
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Table 4-3. Comparison of comDlctenes^ of 9-) 13- » and 
■ 17-yeat-Qld student lists 



- 


..■9-year-olds 


13-year-olds 


17-year-ol(is 


Tear of check 


Year 04 


Year P6 ^^ 


Year 07 


Huinber of DSs checked 
Huinber of schools in sample 


28 


28 ^ 


12 




39 _ 


41 


Completeness of stucfent ^fraiae" 


.975 


..947 ' ■- 


.986 ■ 


Humb'er vof students, checked 


2,356 


6,450 


• 9,556 


Estiinatfid standard error^ 


.007 


.oV? * 


I- ' 

.033 


Huinber of school units us^d 








to estimate standard error 


5 - 


5 

1 


10 



. N • - ■ ■ . 

Table 4«4. Large errors in SLF preparation tabulated by^ region 





Region^ 


Humber of schools with percent 
jiifference greater than 5% 


Total 
number of schools 


■ 


Korthe^st 




11 


* • 


Southeast 




9 

IT 




Central 








^West \ 


2 . .X 

■ 1 


9 ' 




Total ' ~ 


• 7 ■ 

* ^ ' ' ^ 


• 40 



■ ■ 
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vocational-technical students were classified as ineligibles by the Dis-' 

^ f • " 

trict Supervisor. The error was* corrected in RTI's records and the DS was 
mfomted of* the, departure from specifications. Anotfi^ax DS used an in- 
correct subsamplmg interval to compute total eligibles. This error was 
also corrected in RTj[*s records and the DS"- informed.* 

Clerical' errors" in saaiple selection were noted in the following 
schools: 



. PSU School 



PSU School 





■5. SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS 

, • . . 

^Data were collected froip a probability sample of Year 11 schools %o ^ 

• ■ f 

detettaine,*.^ar^racy of trans^fer of SLF data, adhbrence by field st^ff to 

package adaiaistration specifications and sampling procedures, and com- 

pleteness of student lists. ' 

The following general, conclusions are offere'd: ^ — 

No ser4i:$us atiministrative errors wei^e revealed by the quality 
check. • • ' * - 

(B) 'a ratio estimate of'^thci proportion, of students reported in*saiflple 
: Nschools .for Yean 11 was 0.984. The standard error associated 

with this estimate was .0046. 

following recommendatioas are offe^^ed: 

(A) It should be made. clear to the DSs th^t any deviation froth pre- 
scribed procedures without the authori2:ation' of their supervisor 
. may result \m termitiation. ' 

' (B) Quality checks should continui^ to be performed by RTI's National 
Assessment Administration Center. ^ 

(C) Finally, a probability sa^le of schools at all three age cUsses 
should be selected for future Equality chec'ks.' It is also re^'com* 
mended that the sample be designed to mej^t tte *f ollpwing require- 
ments: - 

/ ' ^ . ' ' 

1. At least one school supervised^by each^District Supervisor 
^at each age class 'should be ilacluded *in .the quality check 
sample4 . " , / ^ ^ 

' 2. A 'ratio estimate o'f.the completeness of the student sampling 
frame' should be obtained across all age classes; CT^ 

%. . Estimates of the ^variance of the ratio^ estimates in item ^2 
* above can;be obtained."*. ^ ^ ^ 

Within each selected school, the following items should be checked: 



1. Cosqpleteness of the student sampling frame; 

2. Verification that the student sample was properly selected; 

3. Select^ion of a^ ^ndom sample of* the regular and fcflloVup 
packages In each quality check school and selection of thtee 
respondents -foe each of Jthese packages JEof two purposes; 



.-221' . 



a. ' Verificatioa that these selected students did, in 'fact, 
take packages; 

i. Verification that grade? fex, ^and birthdate information 
y^zt " properly tr^aascribed/^f roA Administration Schedule 



to package 'for ea^h of . tblese students. 
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: APPENDIX B 
Quality Check Forms 





AT* 




B-2 



"1^ QUALITY CHEtK 



•3 



. PSU miMBER 



*AHE OF SCHOOL: 



SCHOOL NUMBER 



T 



-f— 



\ 



PHONE HUHBER: 



RINCIPAL/QOORDINAIOJ/: _ 
"n^HONE COKTACX: * . 

* ' - 

1. Explain purpose of Quality Check. 

■* 

2. Estimate "length of time necessary fb^ check. 
2\ Insure SLF storage envelope is avail-abler 

' 4. -Arrange an appointnent for Quality Check. 
JPOnriMEIT SEX FOR: ^ : — 



' Does jrour family 2«t a news pap ejr^n gularly ? ' 

y*s <=> lUr c=>.ldon'tJ<now/. 
Does your (amtly %ti ziy masazines regularly? 

CD .Yes ' C2J No c=» I don't know. 
Are khere mote than 25 books in your home? ^j. 

^ ■ * 

<=>X«s' <=> No 'x=) J don't know. 
. Is there an encydopedia in ydjir home? 

<=> Yes No - o I don't imow. 

Ho\r raBch school did your ^atiler complete? ' 
/ (FILL IN THE QHE OVAL whichJaest stows how much school your 
; father completed.) 

<=>■ Did not complete the 8th grade '• '. 
Completed the 8th grade, but did not go to high school 
Went to high spifool, but did not graduate from high school ' 

CD Gnduate^ from hi^i_schoolJ . - 

<^ Some educattbnf after graduation froro high school " 

- O I don't, know. ^ ^ 

How much school did youf naothsr cor^pIeM? \ 
<FItL X& THE O^TE OVAL whicTbcst shows how much school your 
motfaerjcompbted.) 

CP Did not complttt tht 8th grade. 

G Compltted tht 8th gnde, but did not go to high school 

C3 Went to bfgh school, buMd not graduate froii high sch6oI , . 

• t 
GraSoated from high school 

Some education after graduation from high ^hoof 

0 ^ 

Idon*tknow*- ^ * . , 

. > Where .did you lire on your ninth birtWay? 

CD In- the United States (Please specify the state or territo^.) . 

■ • 

CD Outsiflt of tht Udltid 3Utes (Pteis« sptci^/ the country.) 



DO KOT^RtTX 
IKTHC 4" 

ae^ow. 



: ARtA 




AZ 
AH 
CA 
CO 

er 

DE 
FL to 
GA 
HI 
ID 
IL 
IN c=> 
lA (=) 
KS <=> 
KY<=> 
LA<=> 
ME CD 

m<=> 

MA<=> 
m <=> 
MNC3 
MSc=> 
MOc=> 
MTc=> 
NBc=> 
NVC=J 
NHc=> 
NJ <=> 

m<=> 

NYto 
NCc=> 
NDc=> 
OHc=> 
OKC=) 
ORc=> 
<=> 

SC (=) 
S 

IX 
UT" 
VT 
VA 
WAc=5i 
WV 
\VI 
WY 
DC 



If 



i BL CD 



1^^ ' • 



ERJCJon'tlmow. 
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n.cCDi 
ref.c=>| 



J) 



A. 



Appi-oval Hxp^es .9/30/^1 



NATIONAL ASSESSMENT OF ^UCATIONAL PROGRESS 
Y>eat 13 Age Class 



PSU Mo. 



:hool Ho. 



A8ses8mehc Verification 



Package Ho. 



Supervisor ^ 



Date 



If a student Is absent f roa school on day of jQuallcy Check select )an . 
additional student using the random nmtber table above. \ 



1. Do yo4j reneaber participating In National Assessment? (If Ho, 
describe assessnent briefly and probe}.- ' 



COMientsf" (1) 
(3) 



2. ^as a test erd«lnldtrator%resent throughout .the te^t? 
^ Cements: <*) 

(2) 



3. Were there any teachers or 'a principal In the ro'on during the 
Cestlng? 

, Coments: (I) ' ^ i 

(2) ■ 

(3) -L - 

I 34 . • ' " ■ 

h£SI£^ Ho. 23^1-41 ' . . ' 




ID HO. 


ID HQ^ 


ID HO, 


Yes 


Ho 


fes 


' Ho 


Yes 


Ho 






*• 






i- 




LK_>r^ (2) (2 




Yes 


Ho 


Yes 




Yes 


Ho 






• 


v 

'-I 







tic 

I 





) m (2 


[} , 


\ Yes 


Ho 


Yes 


Ho- 


Yes 


Ho 








u 







^4 



*» 3 5 



4. Here (he Instructions clear so you fcnew what tp do? 

« • "~ * . 
Coflunentsi (1)- ^ >_ 

(2) ' 

(3) __> * 



5* Here you asked questions such as*. Does your |aally get.- a 
newspaper regularly? (Sitow copy .of Background- Questions) . 

Comments; (1) '• ^ ^ 

(2) • ' ' 



(1) 



6« Could you hear the tape recorder, froa where you were sitting? 
Coouents: (1) [ ; 

. (2) ' > 



(3) 



7« Did the person conducting the test help you with any questions* 
while the tape was playing? 

Conaents: Kl) ^ 

(2) 



(3) 



8. Approximately^ how many students took the test Vlth you? 



Conments: (1) 



« • - (2> A„ 
(3) _* 



• 

♦ 



: ^ 36 * . . 
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SAMPLE SELECTION >CEECmG PROCEDURES 



PSU NO. 



SCHOOL NO* 



Dlrectlo^^j— £xamlne ^asple selection^ package assignment » and transcribing 
procedures for first 5 selected respondents* Dqcuaent each type of error 
In space provided below. Record number of errors on first line of "Cymulated 
Errors" column* ,If the number of cinrnilated errors^ Is l^ss than, or equal to 
the corresponding acceptance number » then stop* Otherwise » examine the 
method tised to select the next 5 Individuals* Document the types of errors. 
Record number of cuioulated errors* If the number of cumulated errors is 
less than or equal to the corresponding acceptance number » then stop. Other** 
vlse» examine the method used to select the next 5 individuals » etc* 
after method used to select at moat 20 respondents h^s been e x a m ined. 



Cumulated 
Sample Slze ^ 



t Cumulated 
Errors 



Acceptance 
Number 



5 
10 
15 
20 



0 

2 
4 
6 



Types of Errors 



FOR CSTTRAL STAFF USE OKI^Y: 



Does sample appeir to hav€^ been selected^properly? 



3-7 



Approval ^Ixes 9/30/81 



^OTAL ELIGIBLES 



FSU NO. 



sCbool no. 



i 



1. Ity in£e>cmation states the SLFs were coopleted from (source) 



' . Can you tell me briefly the procedures used 

completing the SLFs? 



2« H7 information states that the enrollment in your school at the 
beginning ot the yeai^ was as follows: 

1 2 3 4^ ^ 5 6 7 B ' 9 10 U 12 



(Grade Enrollaeiit a& taken from PQ) ' ' 

das there been a significant change in the enrollment figures for any ^ 

grade? (If "Tes" for any grade, cross out original number and 

» - • 

enter current figure.) 



FOR CENTRAI. STA:^ USE OKL?: 



i 



Comments co'ncemlng the source of your check: 
SL7; who completed -SLF?) 



(i.e., computer list or 



HIE Form Ko. 2371-4^ 



-3" 



' ' PSU NO. 



' B-8 

A 

1:0TAL EL'XGISLES 

TOTAL ELIGIBLES ON WORKSHEET 



Approval Expires 9/30/§i 



SCHOOL NO^ 



ALPHABET SECTOR 



Directions; If an alphabet ^sector ,1s provided above » use the school records to 
deteroxine the nuslber of eilglbies^ whos^ last name starts w±th those letteiss 
Included In the alphabet sector* If an alphabet^ sector IsTnot provided , deter* 
,miiie th^ total nunlber of ellglbles ^slng school records* Usfc column 2 below 
to obtain 'the total number of eliglbles by grade* Enter lil c^^l-umn 3 the 
number of eXlglbles on the SLF^ b;^ grade* Enter the sum of^^ese counts for 
columns 2 and 3 In TOTAL row* Multiply TOTAL by factor which has been provided* 
Enter product^as GfiAND TOTAL » Xompare GRAND TOTAL entries/^ columns 2 axid 
3 with total eliglbles entered above^ Resolve any discrepancies* Explain 
discrepancies in space provlde^^* ' . 



(1)^ 

Grade 


• , (2) 
Number of Eliglbles , 
. . .in Records 


(3) \ 
Number of Eliglbles \ 
on SLF \ 




• * * 

Tot^* « 


* 

Total - 


2 * ' 

• 


Total - 


Total » 


3 


* 

/ Total ■ 


^Total - 


"A 


• 

■ 

m 

' Total « 


Total - 1 


' 5 ' 

• 


1 * *Total ■ 


• 

1 

Total " 


6 

_FRir • 


* 

Total * 


• 

• Tot'al - 



2371-42 



,40 -V 





Grade 
* « 


' V (2) , • 
- J ' Number of Eligiblea . 
* , , ^ in Records 


X (3) 

Ntm/er of Eligibl^s 
*jv^ • on SLF ' 




* • 

" . * Total - 

, 


-^^^K 

' Total ■ 


. 8 . 


f Total « 


, \ Total » 


9 
• 

• 


* local » 


Total ■ 


10 ' 


■c- 

\^ ^ Total • 


t 

Totaj?^ . ^ • 


11 — 


1 

^ Total « * 


i 

% 

^' ' Total *• 


• 

••12 ' * 

> 


Total. • 


■ft 

• Total • 


total' ' • 






Multiply 
TOTAL; byi ^ 




• * 


.GBim TOCAL 


4 




fiiplanatlpn for discrepances: ^ . ^ ' • > 
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YEAR 11 ESTIMA.TION PROCEDDRES FOR SLP COMPLETION RATES 

' ' , V 

The Year 11 quality check sample design utilized a one-stage cluster 
sampling procedure within each stratum* The sampling units were either 
^ single schools or a relatively large school (in terms of age class 
' enrollment) and a nearby standby school or a split session,- 

In the subsequent discussion, the following definitions are required: 

^ 

iQh) = the number of ellgibles reported from the quality check 
school'of District Supervisor stratum-h and age class-j; 

Y(jh) = the number of ellgibles reported on the Worksheet of the 
school from District Supervisor stratum-h and age class-j; 

n(jh) s the number of nonstandby sample school^ selected from District 
^ Supervisor stratum-h and age class-j ; 

H(1h) 5 the number of nonstandby schools contained in District Suppr- 
.vGor stratum-h and age class-j; 

L(j)A = „j;he_tLotaX number of District Supervisors by age class-j 

* strata, , 

A combined ratio estimate of the proportion of the ellgibles which were 

reported by the sample schools can be obtained by ^ ► 

RCj) - ^ 



where ' 



L(j) 
h-l 
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the variance of jibese estimates was obtained bj coll(ppsing tbre thtee age 
class strata vltSin each Disti;ict Supervisor stratum:. <^ 



t (i). var t (i) 2 cov [t^(i)jt^(i)l 

var Rcm It^i '{ : ^ 2 — TTX - / t (1) t (1) — 



7 * • . 



where ^ 



4 > 



^ j H(h)tll(h) - nft)! j j2(jj 



ft 



COV 

and HfaerA , \ > 



In the preceding and subsequent formtila: 



nCh) ■ the number t>£ nonstanjlby sample schools selected from 
« \ District Supervisor stratum-h; 

N(h) ■ the nuznber of nonstandby scSbols contained ^ District 
Supervisor stratua-h; 

H ■ the numbet of ^District Supervis^x strata. 



2 2 

The estimates for s (i) , s^(i), and s_<i) were obtained by 
averaging measur.e^yif variance and covarlanc^ over the H District 
Supervisor st^ta> 



y H n(h)-l 



1 



s^(i) - ^2 \ -a(h)] . 



and 



h"l j"l 



where 



n(h) 

E X(jh) ■ . 



9 




n(h) 



^ n(h} 



7(h) - ^ 



r iqh) 



- n(h) 

» * 

Table C-1 shows, vkiues pf H(jh) , n(jh) , X(jh), and Y(jh) for each 
Stratum In the Year ij^data. " ^ 
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